[SPARK-38214][SS]No need to filter windows when windowDuration is multiple of slideDuration by nyingping · Pull Request #35526 · apache/spark

nyingping · 2022-02-15T09:00:40Z

What changes were proposed in this pull request?

At present, the sliding window adopts the form of expand + filter, but in some cases, filter is not necessary.

Filtering is required if the sliding window is irregular. When the window length is divided by the slide length the result is an integer (I believe this is also the case for most work scenarios in practice for sliding window), there is no need to filter, which can save calculation resources and improve performance.

Why are the changes needed?

save calculation resources and improve performance.

Does this PR introduce any user-facing change?

NO

How was this patch tested?

UT and benchmark.

simple benchmark in this commit ,thanks HeartSaVioR@d532b6f

-------case 1

spark.range(numOfRow)
.selectExpr("CAST(id AS timestamp) AS time")
.select(window(col("time"), "15 seconds", "3 seconds", "2 seconds"))
.count()

Result:

Java HotSpot(TM) 64-Bit Server VM 1.8.0_291-b10 on Windows 10 10.0
AMD64 Family 23 Model 96 Stepping 1, AuthenticAMD
sliding windows:                          Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
old logic                                           799            866          70         12.5          79.9       1.0X
new logic                                            58             68           9        171.2           5.8      13.7X

-------case 2

spark.range(numOfRow)
.selectExpr("CAST(id AS timestamp) AS time")
.select(window(col("time"), "10 seconds", "5 seconds", "2 seconds"))
.count()

Result:

Java HotSpot(TM) 64-Bit Server VM 1.8.0_291-b10 on Windows 10 10.0
AMD64 Family 23 Model 96 Stepping 1, AuthenticAMD
sliding windows:                          Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
old logic                                           359            409          52         27.8          35.9       1.0X
new logic                                            47             54           4        212.6           4.7       7.6X

improve structured streaming window of calculated

nyingping · 2022-02-15T11:15:35Z

I'll check it later

HeartSaVioR

Seems OK.
@viirya Would you mind validating this fix as a follow-up of #35362? Thanks in advance!

sql/core/src/test/scala/org/apache/spark/sql/DataFrameTimeWindowingSuite.scala

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala

viirya · 2022-02-16T01:41:53Z

Is this a follow-up of #35362? Looks like a different one. But seems okay. Will re-check it later.

HeartSaVioR · 2022-02-16T01:51:41Z

Yeah I meant additional optimization along with previous one. Sorry if I confused you.

…PARK-38214

AmplabJenkins · 2022-02-16T05:47:54Z

Can one of the admins verify this patch?

nyingping · 2022-02-16T05:50:48Z

Sorry,it's my fault.I mixed the update history of the branch of the previous with the present, caused interference and misunderstanding.

sql/core/src/test/scala/org/apache/spark/sql/DataFrameTimeWindowingSuite.scala

HeartSaVioR

+1 pending tests. Thanks for the contribution!

sql/core/src/test/scala/org/apache/spark/sql/DataFrameTimeWindowingSuite.scala

HeartSaVioR · 2022-02-17T06:08:10Z

I'll leave this in a day to see the chance of another reviews from others. I'll merge this tomorrow if there's no new feedback.

HeartSaVioR · 2022-02-18T01:15:48Z

OK, no feedback on working hour in US timezone.

Thanks! Merging to master.

HeartSaVioR · 2022-02-18T01:18:17Z

Thanks @nyingping for the contribution! I merged into master.

nyingping · 2022-02-18T01:22:35Z

@HeartSaVioR Thank you for review very much!

nyingping added 16 commits January 29, 2022 12:33

improve structured streaming window of calculated

1df0cc8

improve structured streaming window of calculated

8bf8e65

improve structured streaming window of calculated

d8d0799

improve structured streaming window of calculated

530c5b8

improve structured streaming window of calculated

f0e0ee8

improve structured streaming window of calculated

91c7e45

Merge branch 'apache:master' into main

c839ac2

Update Analyzer.scala

118c026

improve structured streaming window of calculated

Merge branch 'apache:master' into main

fabd761

remove the overlappingWindows parameter of the getwindow function

625b4eb

Merge remote-tracking branch 'origin/main'

333b92e

remove the overlappingWindows parameter of the getwindow function

1eaf6e5

Merge branch 'apache:master' into main

55f5dae

No need to filter data when the sliding window length is not redundant

e13ecac

No need to filter data when the sliding window length is not redundant

f936043

No need to filter data when the sliding window length is not redundant

3f179a2

github-actions bot added the SQL label Feb 15, 2022

Merge branch 'apache:master' into SPARK-38214

d1e3517

HeartSaVioR reviewed Feb 15, 2022

View reviewed changes

sql/core/src/test/scala/org/apache/spark/sql/DataFrameTimeWindowingSuite.scala Outdated Show resolved Hide resolved

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala Show resolved Hide resolved

nyingping added 3 commits February 16, 2022 10:57

add code comment and fixed the test case

17cd212

Merge branch 'SPARK-38214' of https://github.com/nyingping/spark into S…

be7effe

…PARK-38214

add code comment and fixed the test case

491de3b

the simple "import" fix

1e48f82

HeartSaVioR reviewed Feb 16, 2022

View reviewed changes

nyingping changed the title ~~[SPARK-38214][SS]No need to filter data when the sliding window length is not redundant~~ [SPARK-38214][SS]No need to filter windows when windowDuration is multiple of slideDuration Feb 17, 2022

updated the test case

198d9c3

HeartSaVioR reviewed Feb 17, 2022

View reviewed changes

sql/core/src/test/scala/org/apache/spark/sql/DataFrameTimeWindowingSuite.scala Outdated Show resolved Hide resolved

HeartSaVioR approved these changes Feb 17, 2022

View reviewed changes

nyingping added 2 commits February 17, 2022 10:57

delete empty line

35a9704

simply space fix

f40aa0f

HeartSaVioR reviewed Feb 17, 2022

View reviewed changes

sql/core/src/test/scala/org/apache/spark/sql/DataFrameTimeWindowingSuite.scala Outdated Show resolved Hide resolved

simply space fix

72485f6

HeartSaVioR closed this in e86c8e2 Feb 18, 2022

Conversation

nyingping commented Feb 15, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

-------case 1

-------case 2

Uh oh!

nyingping commented Feb 15, 2022

Uh oh!

HeartSaVioR left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

viirya commented Feb 16, 2022

Uh oh!

HeartSaVioR commented Feb 16, 2022

Uh oh!

AmplabJenkins commented Feb 16, 2022

Uh oh!

nyingping commented Feb 16, 2022

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

HeartSaVioR left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

HeartSaVioR commented Feb 17, 2022

Uh oh!

HeartSaVioR commented Feb 18, 2022

Uh oh!

HeartSaVioR commented Feb 18, 2022

Uh oh!

nyingping commented Feb 18, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

nyingping commented Feb 15, 2022 •

edited

Loading